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I .   INTRODUCTION 

A.   GENERAL 

Comparisons  between  operational  field  tests  and  high 
resolution  combat  models  serve  a  dual  purpose.  The 
operational  testers  can  save  millions  of  dollars  due  to 
improved  planning  and  evaluation  provided  by  the  judicious 
application  of  the  models.  The  modeling  community  can  enhance 
and  improve  their  models  based  on  real  world  occurrences  in  a 
field  test.  The  Army's  Model-Test-Model  (M-T-M)  concept  is 
the  integration  of  operational  field  tests  and  combat  models. 
This  thesis  contains  a  report  of  our  effort  to  analyze  the 
feasibility  of  accrediting  the  Janus  (A)  high  resolution  combat 
model  for  the  post-test  modeling  of  tank  engagement  ranges. 

We  used  data  from  the  Line  Of  Sight-Forward-Heavy  Initial 
Operational  Test  (LOS-F-H  lOT)  conducted  at  Fort  Hunter 
Liggett,  California  from  April  through  May  1990.  First,  an 
automated  process  to  replicate  the  field  test  vehicle  routes 
in  Janus  (A)  was  developed.  Second,  the  test  data  were 
analyzed  to  see  if  the  data  were  adequate  to  support  a 
comparison.  Third,  where  appropriate,  comparisons  were 
conducted  between  the  test  data  and  model  output.  Appendix  B 
contains  a  list  of  acronyms  used  in  this  thesis. 


B.  DEFINITION  OF  ACCREDITATION 

Before  an  assessment  of  the  feasibility  of  accrediting 
Janus (A)  for  analysis  of  tank  engagement  ranges  is  made,  we 
need  to  define  accreditation.  According  to  a  memorandum 
signed  by  Mr.  Walter  W.  Hollis,  Deputy  Under  Secretary  of  the 
Army  (Operations  Research)  ,  on  October  30,  1989,  accreditation 
is  "Certification  that  a  model  is  acceptable  for  use  for  a 
specific  type(s)  of  application (s) . "  [Ref.  1]  Based  on  this 
definition,  the  thrust  of  this  thesis  is  to  determine  if 
Janus  (A)  is  acceptable  for  use  for  the  specific  application  of 
modeling  tank  engagements  that  occur  at  Fort  Hunter  Liggett 
during  a  field  test.  In  other  words,  can  we  rely  on  Janus (A) 
to  reasonably  approximate  tank  engagement  ranges  that  occur  at 
Fort  Hunter  Liggett  during  a  field  test?  Validation  is  "The 
process  of  determining  that  a  model  is  an  accurate 
representation  of  the  intended  real-world  entity  from  the 
perspective  of  the  intended  use  of  the  model,"  [Ref.  1] 
Validation  of  Janus (A)  is  beyond  the  scope  of  this  thesis. 
However,  our  work  to  accredit  Janus (A)  supports  the  Army  goal 
of  validating  Janus (A). 

C.  MODEL-TEST-MODEL  (M-T-M)  CONCEPT 

Although  the  M-T-M  concept  has  been  used  in  the  Army, 
there  is  no  formal  Army  publication  that  clearly  defines  the 
concept.  US  Army  Training  and  Doctrine  Command  (TRADOC)  at 
White  Sands  Missile  Range,  New  Mexico  and  Research  Analysis 


and  Maintenance,  Inc.  have  conducted  most  of  the  work  in 
M-T-M.  In  October  1990,  Mr.  Mollis  asked  TRADOC  Test  and 
Experimentation  Center  (TEC)  at  Fort  Hunter  Liggett, 
California  to  work  to  improve  the  M-T-M  methodology  [Ref.  2]. 
TEC'S  first  steps  were  to  enlist  the  support  of  TRADOC 
Analysis  Command  (Monterey)  and  to  jointly  define  the  concept 
as  applied  to  operational  testing.  Their  joint  briefing  forms 
the  basis  for  our  description  of  the  M-T-M  concept,  and  is  our 
guiding  definition  of  the  concept  [Ref.  3]. 

The  three  phases  of  the  M-T-M  concept  are  pre-test 
modeling,  field  test,  and  post-test  modeling.  Since  the  focus 
of  this  thesis  is  on  post-test  modeling,  the  first  two  phases 
are  described  only  briefly. 

1.  Phase  I  (Pre-Test  Modeling) 

The  goal  in  this  phase  is  to  use  a  model  to  help  plan 
the  test  design,  including  recommending  scenarios  for  actual 
use  on  the  ground.  Although  this  should  never  replace  on-the- 
ground  planning,  the  modeler  could  save  many  valuable  hours  by 
conducting  simulations  with  different  force  sizes,  scenarios, 
and  tactics.  The  modeler  could  make  recommendations  for 
improving  the  test  design,  in  terms  of  measures  of  performance 
relevant  to  the  tester. 

2.  Phase  II  (Field  Test) 

The  initial  phase  of  a  field  test  is  usually  the  Force 
Development  Test  and  Evaluation  (FDTE) .   In  this  phase  the 


force  structure,  scenarios,  and  tactics  developed  by  the 
combat  model  in  the  pre-test  modeling  phase  could  be  used  on 
the  ground  in  the  initial  field  test  trials.  The  second  phase 
of  the  field  test  is  often  the  Initial  Operational  Test  (lOT)  . 
Although  the  tactics  must  be  unscripted  for  this  phase,  the 
force  structure  and  scenarios  adopted  during  the  pre-test 
modeling  can  be  used. 

3.  Phase  III  (Post-Test  Modeling) 

There  can  be  many  uses  of  a  model  in  this  phase,  but 
we  will  outline  three  that  we  consider  to  be  the  most 
significant. 

a .  Compare 

The  model  is  set-up  to  replicate  a  specific  field 
test  trial.  We  explain  a  process  to  do  this  for  the  Janus (A) 
combat  model  in  Chapter  III.  The  modeler  runs  the  model  and 
specific  measures  of  performance,  such  as  engagement  ranges, 
are  collected.  The  tester  and  modeler  compare  both  sets  of 
outputs.  The  results  of  this  analysis  lead  to  the  next  stage. 

b.  Improve 

(1) Model .  If  the  results  from  the  model  do  not 

agree  with  the  results  from  the  test,  the  modeler  should  take 
a  hard  look  at  the  way  the  model  attempts  to  replicate  the 
test.  There  may  be  problems  with  the  algorithms  the  model 
uses.  There  may  be  an  error  in  model  input  parameters,  such 
as  vehicle  speed  or  weapon  ranges.  The  modeler's  focus  should 


be  to  adjust  the  model  to  replicate  the  conditions  of  the 
field  test,  not  to  tweak  the  model  just  to  generate  similar 
outputs.  The  modeler  should  make  adjustments  until  he  is 
satisfied  that  the  model's  input  parameters  mirror  the  test 
conditions.  If  the  outputs  from  the  test  and  model  are  still 
different,  he  should  focus  on  the  terrain  database  and 
algorithms  within  the  model  for  possible  improvements. 

(2) Test.   It  may  be  a  mistake  to  conclude  that 

the  model  is  in  error  just  because  the  model  and  test  outputs 
are  different.  Even  though  the  tester  takes  great  care  to 
insure  he  records  all  measurements  accurately;  the  inherent 
nature  of  trying  to  record  physical  occurrence  on  the 
battlefield  is  extremely  complex.  The  resulting  data  are  not 
without  flaw.  Due  to  a  large  amount  of  human  factors  and 
instrumentation  problems,  the  data  may  be  unsuitable  for 
comparison  with  output  from  a  model.  If  the  test  data  are 
seriously  flawed,  the  tester  might  attempt  to  reconstruct  some 
of  the  data.  He  should  review  the  procedures  used  to  collect 
the  data  during  the  test  and  improve  them  for  future  trials. 
We  should  note  that  there  are  varying  degrees  of  simulation 
during  the  field  test  itself.  For  example,  TEC  uses  random 
numbers  to  decide  whether  a  vehicle  engaged  by  another  vehicle 
is  "killed".  If  we  were  to  compare  the  number  of  kills 
between  the  test  and  Janus (A) ,  we  should  be  aware  that  we  are 
comparing  two  simulations.   Additionally,  due  to  the  factors 


described  above,  most  of  the  test  measures  of  effectiveness 
have  large  variances.   These  large  variances  may  lead  the 
analyst  into  accepting  null  hypotheses  of  no  difference  with 
a  very  low  degree  of  power. 
c.  Expand 

If  the  outputs  from  the  model  and  test  are  similar, 
it  may  be  feasible  to  accredit  the  model  for  a  specific 
application.  It  may  also  be  feasible  to  extrapolate  the  test 
evaluation  by  running  the  model  with  force  structures  and 
scenarios  that  could  not  be  finished  during  the  field  test  due 
to  resource  or  testing  constraints.  Extrapolation  of  test 
results  becomes  less  reliable  the  greater  the  changes  to  the 
force  structure  and  scenarios. 

D.   MODEKENG  TANK  ENGAGEMENTS 

1.  Dependency  of  Multiple  Engagements 

We  define  a  multiple  engagement  as  the  tank  firing  at 
the  same  target  within  a  specified  time  interval.  This  time 
interval  is  defined  in  subsection  two.  We  have  observed  that 
ranges  in  a  multiple  engagement  are  related.  The  cause  of 
this  dependence  can  be  attributed  to  the  spatial  relationship 
between  the  firer  and  the  target;  both  the  firer  and  target  do 
not  move  very  far  during  a  multiple  engagement.  The  ranges  in 
a  multiple  engagement  are  thus  statistically  dependent.  We 
show  statistical  evidence  of  this  dependence  in  Chapter  II. 


We  model  the  times  between  rounds  in  a  multiple  engagement  and 
call  them  the  inter-fire  times. 

2.  Inter-Fire  Time  (IFT) 

The  tank  engagement  process  is  extremely  complex.  The 
following  model  is  a  simplification  of  this  process  to  help 
define  the  inter-fire  time  (IFT) . 

As  a  tank  traverses  the  battlefield,  the  tank  commander  is 
searching  for  targets.  Once  he  identifies  a  target,  he  lays 
the  gun  tube  onto  the  gunner's  sight  picture  (lay  time) .  The 
gunner  aims  at  the  target  by  placing  the  crosshairs  of  the 
sight  onto  the  target  (aim  time) .  The  gunner  fires  a  round  at 
the  target.  Time  of  flight  (TOF)  is  the  time  it  takes  for  the 
round  to  reach  the  target  area.  The  tank  commander  then 
decides  if  the  target  was  destroyed  (assess  time) .  For 
subsequent  rounds  fired  at  the  same  target,  the  gunner  aims  at 
the  target  while  the  loader  reloads  the  gun.   The  equation 

JFr=max  {AIM,  RELOAD)  +TOF+ ASSESS 

defines  the  time  between  firings  in  a  multiple  engagement. 
For  subsequent  rounds  in  a  multiple  engagement,  lay  time  is 
zero  because  we  assume  the  gunner  already  has  the  target  in 
his  sight  picture.  The  IFT  includes  the  maximum  of  aim  and 
reload  time  because  these  two  events  occur  simultaneously  on 
subsequent  engagements. 


3.  First  Range  of  Engagement  (FRE) 

The  first  range  of  engagement  (FRE)  is  the  initial 
range  at  which  the  gunner  shoots  at  a  target  in  a  multiple 
engagement,  or  the  only  range  in  a  singular  engagement. 
Figure  1  summarizes  the  engagement  process.  We  conduct  our 
statistical  analysis  with  the  FRE's  rather  than  all  ranges. 
By  considering  only  the  FRE's,  we  are  removing  a  source  of 
dependence  between  engagement  ranges.  We  do  not  claim  that 
our  samples  of  FRE's  are  completely  independent,  but  we  have 
a  better  case  for  using  parametric  and  non-parametric 
statistical  tests  in  analyses  of  these  ranges.  Due  to 
problems  with  the  test  data  outlined  in  Chapter  II,  we  have 
narrowed  the  scope  of  analysis  to  tank  FRE's. 
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Figure  1  Diagram  of  Engagement  Process 
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II.   OPERATIONAL  FIELD  TEST 

A.   BACKGROUND 

The  Line-of-Sight  Forward  Heavy  (LOS-F-H)  is  an  air 
defense  system  armed  with  surface-to-air  missiles  mounted  on 
a  modified  Bradley  Fighting  Vehicle  chassis.  The  mission  of 
the  LOS-F-H  is  to  defend  Army  heavy  divisions  against  air 
attack.  The  purpose  of  the  LOS-F-H  lOT  was  to  determine  the 
operational  effectiveness  of  a  LOS-F-H  platoon  to  accomplish 
its  mission.  TEC  conducted  the  test  at  Fort  Hunter  Liggett, 
California  from  April  9  to  May  23,  1990.  The  test  consisted 
of  50  trials.  Each  trial,  approximately  one  hour  in  length, 
was  a  force-on-force  battle  between  instrumented  Blue  and  Red 
mechanized  forces  of  approximately  battalion  strength.  The 
blue  mechanized  forces  consisted  of  four  LOS-F-H  vehicles,  14 
Abram  Main  Battle  Tanks  (MlAl) ,  and  15  Cavalry  Fighting 
Vehicles  (CFV) .  The  red  mechanized  forces  consisted  of  14 
Future  Soviet  Tanks  (FST)  and  10  armored  personnel  carriers 
(BMP)  .  The  M60A3  main  battle  tank  and  the  M113  armored 
personnel  carrier  were  surrogates  for  the  FST  and  BMP, 
respectively.  In  addition  to  the  ground  forces,  there  were 
various  types  of  helicopters  and  aircraft  on  both  sides. 
For  more  information  on  the  field  test  see  [Ref.  4]. 
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B.   TEST  DATA 

1.  Trial  Selection 

The  test  consisted  of  50  trials  conducted  in  four 
scenarios.  The  blue  force  mission  was  either  defense  facing 
North,  defense  facing  South,  offense  facing  North,  or  offense 
facing  South.  The  red  force  conducted  the  opposite  mission 
accordingly. 

An  analysis  of  all  50  trials  is  beyond  the  scope  of 
this  thesis.  To  minimize  the  effects  of  systematically  varied 
conditions  on  engagement  ranges,  six  trials  that  occurred 
under  the  same  conditions,  shown  in  Table  I,  were  selected. 
Table  I   TRIALS  SELECTED 


Trials  Selected 
L099B,L100B,L112B,L122B,L123B,L125B 

Factor 

Condition 

Light 

Day 

Blue  Mission 

Defense 

Blue  Orientation 

North 

Chemical  Equip. 

None 

Smoke 

None 

2.  Data  Collection 

TEC  provided  two  types  of  data,  position  location 
(PLS)  and  battle  (BTL)  files,  for  each  of  the  six  trials 
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selected.  The  PLS  file  contained  a  record  of  each  vehicle's 
location  for  every  second  during  the  trial.  The  PLS  file  was 
used  to  create  the  vehicle  routes  in  Janus (A) .  This  procedure 
is  described  in  Chapter  III.  The  BTL  file  contained  a  record 
of  each  engagement  between  vehicles,  including  the  firer, 
target,  and  associated  engagement  range,  if  available.  These 
data  were  compared  with  the  output  from  Janus (A) . 
3.  Data  Limitations 

a.  Position  Location  Errors 

There  are  errors  associated  with  position  location 
data.  These  errors  have  an  impact  on  vehicle  routes  and 
engagement  ranges.  If  the  position  location  is  inaccurately 
recorded,  the  recorded  vehicle  route  could  be  quite  different 
from  the  actual.  Additionally,  engagement  ranges  are 
calculated  from  the  position  location  data  using  the  formula 

Range=^  iX^i,^,-X,„^^,)  ^^  (  Yfirez'^ target)  ' 

where  X  and  Y  are  the  coordinates  of  the  f  irer  and  target  when 
the  firer  pulls  the  trigger.  If  the  position  location  is 
inaccurate  at  the  time  of  an  engagement,  the  engagement  range 
could  be  inaccurate.  The  range  measuring  system  (RMS)  at  Fort 
Hunter  Liggett  records  position  location  and  engagement  range 
data.  An  explanation  of  this  system  is  given  in  [Ref.  5]. 
The  three  main  errors  associated  with  position  location  data 
are  jitter,  gaps,  and  spikes. 
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(1) Jitter.  This  is  a  slight  error  in  recording 

vehicle  location.  The  vehicle  appears  to  be  moving  to  a  series 
of  nearby  points  when  it  is  stationary.  This  error  is  caused 
by  slight  errors  in  triangulation  by  the  RMS  at  Fort  Hunter 
Liggett . 

(2) Gaps.    These  are  losses  of  a  vehicle's 

location  for  various  reasons.  The  main  cause  is  the  vehicle 
moving  into  an  area  where  its  signal  cannot  be  received  by  the 
RMS. 

(3) Spikes.  These  are  major  errors  in  recording 

vehicle  location.   A  vehicle  appears  to  move  to  a  location 
that  is  clearly  not  logical  given  its  previous  location. 
Inaccurate  triangulation  by  the  RMS  is  the  main  cause  of  this 
error. 

b.  Unknown  Engagement  Ranges 

There  are  several  reasons  for  errors  in  recording 
engagement  ranges.  We  have  described  in  the  above  paragraph 
the  impact  of  inaccurate  position  location  on  engagement 
ranges.  This  section  focuses  on  why  some  engagement  ranges 
are  not  recorded  in  the  data. 

During  the  field  test,  technicians  instrument  weapons 
with  lasers.  When  a  gunner  pulls  a  weapon  trigger,  a  laser 
beam  is  sent  in  the  direction  of  the  aimpoint.  The  firer's 
identification  is  recorded  even  if  the  target  is  unknown.  If 
the  sensors  on  the  target  receive  the  laser  beam,  the  RMS 


12 


calculates  the  engagement  range.  If  the  laser  beam  is  not 
received  by  the  target,  the  RMS  cannot  determine  the  target's 
location;  an  engagement  range  is  not  calculated.  Detailed 
reasons  for  these  unknown  engagement  ranges  are  as  follows. 

(1) Missed  Target.  The  gunner  aimed  incorrectly 

and  legitimately  missed  the  target.  This  error  is  most  likely 
the  cause  of  most  of  the  unknown  engagement  ranges. 

(2) Improper  Laser  Boresicrht.  The  laser  and  the 

weapon  are  not  aiming  in  exactly  the  same  direction.  The 
gunner  correctly  aims  at  the  target,  but  the  laser  beam 
travels  in  a  slightly  different  direction  and  misses  the 
target. 

(3) Improper  use  of  Sensors.  The  laser  reaches 

the  target,  but  the  sensors  do  not  register  because  they  have 
not  been  properly  emplaced  or  for  some  other  reason  cannot 
register  illumination  by  the  firer's  laser. 

(4) Attenuation  of  Laser  Beam.    The  gunner 

correctly  aims  at  the  target,  but  the  laser  beam  does  not 
reach  the  target  because  it  is  attenuated  by  heat  or  dust. 

(5) Insufficient  Power  Output.    The  gunner 

correctly  aims  at  the  target,  but  the  laser  does  not  have 
sufficient  power  to  reach  the  target. 

(6) Buffer  Overload.   In  this  case  the  sensors 

on  the  target  receive  the  laser  beam,  but  buffers  that  act  as 
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temporary  holding  places  for  data  become  overloaded  and  the 
engagement  is  not  sent  to  the  main  computer. 
4.  Scunple  Size  Analysis 

The  limitations  described  above  had  a  great  impact  on 
the  sample  size  of  engagement  ranges.  Tables  II-V  list  the 
number  of  engagement  ranges  recorded  in  the  data  for  each  of 
the  six  trials  we  selected.  These  are  the  total  number  of 
engagement  ranges  before  selecting  only  the  FRE's. 
Table  II   MlAl  ENGAGEMENT  SUMMARY 


TRIAL/ 
TARGET 

L099B 

LIOOB 

L112B 

L122B 

L123B 

L125B 

FST 

37 

38 

29 

22 

24 

13 

BMP 

9 

12 

10 

9 

14 

8 

OTHER 

15 

8 

7 

0 

3 

0 

TOTAL 
KNOWN 
RANGES 

61 

58 

46 

31 

41 

21 

TOTAL 

UNKNOWN 

RANGES 

149 

17 

29 

48 

28 

42 

TOTAL 
#  ENGAG 

210 

75 

75 

79 

69 

63 

PERCENT 

KNOWN 

RANGES 

29.04 

77.33 

61.33 

39.24 

59.42 

33.33 

MlAl  engagement  ranges  have  the  highest  percentage  of 
known  ranges  for  each  trial.  The  average  percentage  of  known 
ranges  for  all  six  trials  is  45  percent  of  all  shots  fired. 
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The  MlAl  ranges  were  compared  with  Janus (A)  data.   The  FRE's 

were  selected  from  the  sample  of  known  ranges.   Chapter  IV 

contains  the  sample  sizes  of  FRE's  used  in  the  statistical 

analyses. 

Te±>le  III   FST  ENGAGEMENT  SUMMARY 


TRIAL/ 
TARGET 

L099B 

LIOOB 

L112B 

L122B 

L123B 

L125B 

MlAl 

4 

2 

0 

3 

2 

16 

CFV 

0 

3 

0 

4 

1 

15 

OTHER 

0 

7 

0 

3 

3 

5 

TOTAL 
KNOWN 
RANGES 

4 

12 

0 

10 

6 

36 

TOTAL 

UNKNOWN 

RANGES 

67 

15 

16 

75 

23 

89 

TOTAL 
#  ENGAG 

71 

27 

16 

85 

29 

125 

PERCENT 

KNOWN 

RANGES 

5.63 

44.44 

0 

11.76 

20.69 

28.80 

The  average  percentage  of  known  FST  engagement  ranges  for 
all  six  trials  was  19  percent  of  all  shots  fired.  With  the 
exception  of  trial  L125B,  the  sample  size  was  less  than  five 
for  each  trial.  This  is  not  a  good  sample  of  the  population 
of  FST  engagement  ranges.  Thus,  we  used  only  trial  L125B  to 
compare  FST  engagements  to  Janus (A) . 
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Table  IV   CFV  ENGAGEMENT  SUMMARY 


TRIAL/ 
TARGET 

L099B 

LIOOB 

L112B 

L122B 

L123B 

L125B 

FST 

13 

21 

4 

7 

8 

50 

BMP 

19 

155 

7 

39 

40 

167 

OTHER 

89 

115 

47 

28 

19 

21 

TOTAL 
KNOWN 
RANGES 

121 

291 

58 

74 

67 

238 

TOTAL 

UNKNOWN 

RANGES 

463 

812 

97 

146 

97 

1190 

TOTAL 
#  ENGAG 

584 

1103 

155 

220 

164 

1428 

PERCENT 

KNOWN 

RANGES 

20.72 

26.38 

37.41 

33.64 

40.85 

16.67 

The  average  percentage  of  known  engagement  ranges  for  the 
CFV  was  23  percent  of  all  shots  fired.  Although  this  appears 
to  be  a  reasonable  nximber,  further  analysis  of  the  known 
engagement  ranges  reveals  that  most  of  these  ranges  are 
repetitive.  The  gunner  usually  fires  several  rounds  in 
succession  at  the  same  target  using  the  25mm  chain  gun.  The 
ranges  of  these  rounds  recorded  in  the  data  are  usually  the 
same.  As  shown  in  Section  five,  the  number  of  FRE's  is  low 
compared  to  the  total  number  of  engagements.  Thus,  a 
comparison  of  these  ranges  between  the  test  and  Janus (A)  would 
lead  to  inconclusive  results. 
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Table  V   BMP  ENGAGEMENT  SUMMARY 


TRIAL/ 
TARGET 

L099B 

LIOOB 

L112B 

L122B 

L123B 

L125B 

MlAl 

0 

3 

1 

0 

2 

3 

CFV 

13 

0 

0 

0 

0 

24 

OTHER 

7 

5 

2 

5 

0 

9 

TOTAL 
KNOWN 
RANGES 

20 

8 

3 

5 

2 

36 

TOTAL 

UNKNOWN 

RANGES 

414 

441 

147 

350 

135 

355 

TOTAL 
#  ENGAG 

434 

449 

150 

355 

137 

391 

PERCENT 

KNOWN 

RANGES 

4.61 

1.78 

2.00 

1.41 

1.46 

9.21 

The  average  percentage  of  known  engagement  ranges  for  the 
BMP  was  only  four  percent  of  all  shots  fired.  This  low  number 
indicates  the  sample  size  is  not  a  good  representation  of  the 
population  of  BMP  engagements.  Thus,  a  comparison  of  these 
ranges  between  the  test  and  Janus (A)  was  not  conducted. 

5.  Correlation  Analysis 

Statistical  evidence  of  the  dependence  of  ranges  in  a 
multiple  engagement  is  shown  through  a  correlation  analysis. 
An  analysis  of  the  MlAl  and  CFV  ranges  yields  very  high 
correlation  between  the  first  and  subsequent  ranges  within  an 
Inter  Fire  Time  (IFT)  of  30  seconds. 
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Table  VI  MlAl  SAMPLE  CORRELATIONS 


Trial  L099B 
Coefficient  (Sample  Size) 

Engagement 
Number 

1st  Range  of 
Engagement 

2nd  Range  of 
Engagement 

3rd  Range  of 
Engagement 

1st  Range  of 
Engagement 

1.0000 
(22) 

.9986 
(12) 

1.0000 
(2) 

2nd  Range  of 
Engagement 

.9986 
(12) 

1.0000 
(22) 

1.0000 
(2) 

3rd  Range  of 
Engagement 

1.0000 
(2) 

1.0000 
(2) 

1.0000 
(2) 

The  initial  sample  size  of  MlAl  versus  FST  engagement 
ranges  for  trial  L099B  was  37.  Out  of  this  number,  22  are 
FRE's,  12  are  second  ranges  of  engagement,  two  are  third 
ranges  of  engagement,  and  one  is  a  fourth  range  of  engagement. 
The  high  estimated  correlation  coefficients  shown  in  Table  VI 
indicate  dependence  between  the  types  of  engagement  ranges. 
Hypotheses  that  the  coefficients  are  zero  are  rejected  at  a 
significance  level  of  .05.  Analyses  of  the  MlAl  engagements 
for  the  remaining  five  trials  yield  similar  results. 

An  analysis  of  the  CFV  ranges  indicates  the  same  high 
correlation  between  the  first  and  subsequent  ranges  in  a 
multiple  engagement.  Additionally,  the  sample  size  of  CFV 
versus  BMP  engagement  ranges  for  Trial  L099B  was  19.  Out  of 
this  number  only  four  are  FRE's.  A  statistical  analysis  with 
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only  four  data  points  out  of  a  total  of  584  rounds  fired  would 
lead  to  inconclusive  results.  Analyses  of  the  CFV  engagements 
for  the  remaining  five  trials  yield  similar  results. 

The  low  percentage  of  known  engagement  ranges  for  all 
vehicles  except  the  MlAl  is  caused  by  a  combination  of  the 
limitations  previously  discussed.  As  a  supplement  to  the  RMS 
system,  TEC  mounts  through-sight  video  on  vehicles.  Video 
data  reduction  techniques  are  then  used  to  reconstruct 
engagements  not  captured  by  the  RMS.  Due  to  lack  of  time  and 
resources,  this  technique  was  used  only  on  the  LOS-F-H 
vehicles.  TEC  is  also  working  on  a  new  system  to  replace  the 
RMS.  This  new  system,  called  KTOPS,  uses  radar  to  identify 
the  impact  of  the  round.  Knowing  the  impact  of  the  round  will 
allow  the  engagement  range  to  be  calculated  even  if  the  round 
fails  to  hit  the  target.  Post-trial  reconstruction  or  the 
KTOPS  system  can  be  used  in  the  future  to  enhance  recording  of 
engagement  ranges. 
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III.   JANUS (A)  COMBAT  MODEL 

A.   INTRODUCTION 

Janus  is  an  event  driven,  high  resolution  combat  model 
named  after  the  Roman  god  who  was  guardian  of  portals  and 
patron  of  beginnings  and  endings.  The  model  runs 
interactively  or  systemically.  In  the  interactive  mode,  the 
red  and  blue  forces  appear  on  separate  video  terminals.  The 
modeler  views  the  battle  as  it  occurs.  He  can  affect  the 
battle  like  a  commander  by  changing  options  such  as  vehicle 
routes  and  speeds.  In  the  systemic  mode,  the  model  runs 
without  a  man-in-the-loop  for  a  specified  time.  Terrain  is 
depicted  with  contour  lines,  vegetation,  and  cities. 
Graphical  symbols  represent  one  or  more  systems.  Each  system 
has  one  or  more  weapons.  For  example,  a  tank  is  a  system 
containing  a  main  gun  and  machine  guns. 

Janus  uses  the  Night  Vision  Electro-Optical  Laboratory 
(NVEOL)  model  for  detection.  When  a  system  detects  a  target, 
an  algorithm  determines  line  of  sight  based  on  terrain  and 
weather.  If  the  system  has  line  of  sight,  can  range  the 
target,  has  ammunition,  and  is  not  in  a  hold-fire  status,  it 
fires  at  the  target.  On  the  graphics  display,  a  red  line  from 
the  firer  to  the  target  designates  a  firing.  The  simulation 
resolves  conflict  by  comparing  random  number  draws  to  a 


20 


probability  of  hit  and  kill  database.  Post-processing  files 
allow  the  analyst  to  collect  a  wide  range  of  data.  For  more 
detailed  information  on  Janus  see  [Ref .  6] . 

B.   TEST  DATA  CONVERSION 

A  challenge  in  converting  data  was  to  take  the  test 
position  location  (PLS)  data  from  each  of  the  six  field  trials 
and  convert  them  into  corresponding  Janus  scenarios.  Without 
such  a  conversion  it  would  be  necessary  to  manually  generate 
the  six  scenarios.  It  is  possible  to  misread  the  many  grid 
coordinates  from  the  field  test  position  location  data  or 
misplace  them  on  the  Janus  screen.  The  human  eye  can  only 
distinguish  six  digits  of  the  ten  digit  grid  coordinates  on 
the  screen.  Additionally,  it  takes  about  six  hours  to  place 
all  the  grid  coordinates  manually.  An  automated  process 
reduces  this  time  to  one  hour. 

Modelers  have  converted  data  from  the  National  Training 
Center  (NTC) ,  located  in  Fort  Irwin  California,  to  run  in 
Janus.  Mr.  Al  Kellner,  a  programmer  from  TRAC  White  Sands 
Missile  Range,  has  written  an  extensive  FORTRAN  program, 
called  INITNTC,  to  convert  NTC  battles  into  Janus  format. 
TRAC  (Monterey)  uses  this  program  to  analyze  NTC  battles  using 
Janus  [Ref.  7],  The  program  takes  four  files  of  NTC  data  and 
makes  the  necessary  Janus  files.  Once  the  Janus  files  are 
created,  the  model  is  run  normally.  For  a  more  detailed 
explanation  of  the  INITNTC  program  see  [Ref.  8]. 
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The  FORTRAN  program  developed  in  this  thesis,  PLSTRN 
(Appendix  A)  ,  takes  the  field  test  PLS  data  file  and 
transforms  it  into  four  pseudo-NTC  data  files.  The  INITNTC 
program  is  then  run  using  these  pseudo-NTC  files  to  create  the 
necessary  Janus  files.  Minor  changes  to  the  INITNTC  FORTRAN 
code  were  made  to  accommodate  differences  in  grid  zone 
designators  between  Fort  Hunter  Liggett  and  Fort  Irwin. 
Records  of  these  changes  are  on  file  at  TRAC-Monterey . 
Figure  2  contains  a  diagram  of  the  conversion  process. 


Field 
Test  Data 
L1  OOb.pIs 


PLSTRN 


NTC  Files 
ntcplay.dat 
ntcmove.dat 
ntcrout.dat 
ntckils.dat 


INITNTC 


Janus  Files 
forces.dat 

system.dat 
dploy.dat 


Figure  2  Conversion  Process 

The  conversion  process  creates  a  scenario  that  replicates 
the  force  structure  and  vehicle  routes  of  a  field  trial.  The 
modeler  can  adjust  several  input  parameters  to  further  enhance 
the  ability  of  the  simulation  to  replicate  a  trial. 
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C.   MODEL  INPUT  PARAMETERS 

1.  Weapon  Characteristics  (Ground  Vehicles) 

Only  the  weapons  used  during  the  field  test  were 
replicated  in  Janus.  For  example,  during  the  field  test,  the 
main  gun  was  the  only  weapon  instrumented  with  lasers  on  each 
MlAl  tank.  Thus,  machine  guns  on  the  MlAl  tanks  were  not  used 
in  Janus. 

Janus  has  a  database  containing  hundreds  of  parameter 
values.  Examples  of  these  parameters  are  weapon  aim  time,  lay 
time,  reload  time,  and  maximum  range.  Unless  specifically 
stated  in  this  thesis,  these  values  were  not  changed  as  we 
assumed  them  to  be  correct.  Subjective  changes  to  these 
parameters  would  decrease  the  reliability  of  our  results. 
Communication  between  the  tester  and  modeler  will  help  to 
identify  the  correct  parameter  values.  Standardization  of 
these  values  is  necessary  to  avoid  modifying  them  to  produce 
desired  test  results. 

2.  Ainmiinition  Basic  Loads 

During  the  field  test,  each  vehicle  had  a  specific 
amount  and  type  of  ammunition  for  each  weapon,  called  an 
ammunition  basic  load.  Testers  limited  the  amount  of  laser 
firings  for  each  weapon  to  each  weapon's  ammunition  basic 
load.  Ammunition  basic  loads  used  in  Janus  were  changed  to 
reflect  the  ammunition  basic  loads  used  in  the  field  test. 
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3 .  Terrain 

In  Janus,  terrain  is  simulated  by  a  series  of 
rectangular  cells.  Each  cell  contains  the  elevation  at  the 
lower  left  corner  of  the  cell  [Ref.  9:  p.  14].  Fifty  meter 
terrain  resolution  was  used  in  the  simulation  because  it 
covered  the  entire  player  maneuver  area  at  Fort  Hunter 
Liggett.  In  fifty  meter  terrain  resolution,  each  side  of  the 
cell  simulates  fifty  meters  of  actual  terrain.  A  vehicle's 
elevation  is  determined  by  interpolation  of  the  elevations  at 
the  four  corner  points  of  the  cell  in  which  the  vehicle  is 
located  [Ref.  9:  p.  15].  Elevation  is  important  because  it  is 
used  by  Janus  to  determine  line  of  sight  between  vehicles. 

Roads  were  added  to  the  terrain  data  base  according  to 
the  Janus  users  manual  using  a  terrain  map  of  the  maneuver 
area. 

4 .  Nodes 

In  the  Janus  display,  a  node  is  a  triangle  connecting 
two  line  segments.  The  line  segments  represent  the  vehicle 
route.  The  node  represents  a  change  in  direction  of  the 
vehicle  route.  A  time  can  be  placed  on  a  node;  a  vehicle 
stops  if  it  reaches  a  node  before  a  time  placed  on  a  node.  A 
vehicle  continues  when  the  simulation  time  is  greater  than  the 
time  placed  on  the  node.  Janus  users  place  times  on  nodes  to 
control  the  timing  of  the  battle.  INITNTC  does  create  nodes 
for  each  vehicle  route;  it  does  not  associate  a  time  with  each 
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node.  These  times  were  placed  manually  for  our  Janus 
scenarios  to  help  synchronize  the  battles.  The  data  for  the 
nodes  were  collected  by  adding  an  output  statement  to  the 
INITNTC  FORTRAN  code.  The  computer  generates  a  file  called 
timenodes.dat  each  time  INITNTC  is  run.  This  file  contains 
the  time  associated  with  each  node. 

5 .  Weather 

The  test  data  included  exact  information  about  the 
visibility,  temperature,  and  other  weather  conditions  that 
existed  during  the  field  test.  These  conditions  were 
replicated  as  closely  as  possible  in  Janus. 

6.  Verification 

The  Janus  verification  program  checks  for  errors  in 
each  scenario.  Failure  to  assign  a  weapon  against  a  target  is 
an  example  of  an  error.  Verification  was  conducted  for  each 
scenario.  All  errors  were  corrected  before  the  simulation  was 
run. 

D.   SIMULATION  RUNS 

The  six  Janus  scenarios  corresponded  to  the  six  field 
trials  selected  for  analysis.  Each  scenario  was  run 
interactively  at  first  to  insure  the  simulation  was  working 
properly.  To  determine  the  number  of  systemic  runs  per 
scenario  for  data  collection,  the  criterion  of  determining  a 
95  percent  confidence  interval  for  the  mean  MlAl  engagement 
range  with  length  100  meters  was  selected.   A  100  meter 
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confidence  interval  is  a  fairly  tight  bound  on  a  mean  that 
could  range  from  500  to  2500  meters.  A  point  estimate  a  of 
the  variance  of  the  MlAl  engagement  range  was  computed  using 
data  from  five  simulation  runs  of  trial  LIOOB.  Solving  for  n 
the  overall  sample  size  of  MlAl  engagements  for  a  (l-ci!)100% 
confidence  interval  of  length  L  in  each  scenario,  using  the 
Central  Limit  Theorem,  yields 


n   =  ^^^ 


(L/2)2 


or 

1.962  .  111559 

n  = "172  . 

502 

The  average  number  of  engagements  per  run  was  18.  To  obtain 
a  sample  size  of  172  engagements,  the  simulation  would  have  to 
be  run  172/18=9.55,  or  approximately  10  times.  Thus,  each  of 
the  six  scenarios  were  run  10  times  for  a  grand  total  of  60 
simulation  runs. 

E.   DATA  COLLECTION 

The  Janus  Analyst  Work  Station  (JAWS)  direct  fire  file  was 
used  to  collect  the  engagement  ranges.  This  flat  ASCII  file 
is  generated  from  the  main  menu  after  a  simulation  run.  A 
FORTRAN  program  computed  the  ranges  from  the  X  and  Y  location 
of  the  vehicles  at  the  time  of  the  engagement  using  the 
equation  described  on  page  11.   A  second  FORTRAN  program 
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computed  the  first  range  of  engagements  (FRE's)  using  an 
Inter-Fire  Time  (IFT)  of  30  seconds.  This  inter-fire  time  was 
based  on  the  values  of  aim  time,  reload  time,  and  time  of 
flight  in  the  Janus  database. 
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IV.   COMPARISON  OF  FIRST  RANGE  OF  ENGAGEMENTS  (FRE's) 

A.  GENERAL 

The  purpose  of  this  comparison  is  to  determine  if  the 
means  and  distributions  of  FRE's  are  the  same  for  the  field 
test  and  Janus.  Samples  from  the  test  consisted  of  FRE's  from 
trials  L099B,  LIOOB,  L112B,  L122B,  L123B,  and  L125B.  Samples 
from  Janus  consisted  of  FRE's  from  ten  simulation  runs  of  each 
of  these  six  trials.  Normality  of  each  of  the  samples  was 
checked  using  the  Kolmogorov-Smirnov  and  Chi-Square  Goodness 
of  Fit  tests.  In  general,  the  data  are  not  normally 
distributed.  Thus,  mostly  nonparametric  methods  were  used  in 
the  comparison.  A  significance  level  of  .05  was  established 
before  conducting  sample  comparisons.  The  statistical 
software,  Statgraphics,  was  used  in  the  analysis  of  engagement 
range  data  [Ref .  10] . 

B.  ASSUMPTIONS 

The  following  assumptions  were  made  before  conducting  Two- 
Sample  statistical  tests. 

1.   Independence  Within  Each  Seimple 

Each  sample  is  a  random  sample  from  its  respective 
population.  The  reason  for  analyzing  FRE's  instead  of  all 
engagement  ranges  was  to  improve  the  tenability  of  assuming 
independence  within  each  sample. 


28 


2 .   Independence  Between  Samples 

Data  in  each  sample  are  independent  of  data  from  other 
samples.  This  assumption  implies  that  the  samples  from  the 
field  test  are  independent  of  the  samples  from  Janus. 

C.   COMPARISON  OF  MEANS 

The  Mann-Whitney  test  was  used  to  test  the  hypothesis 
that  the  FRE  means  are  the  same  between  the  test  and  Janus. 
The  assumptions  for  this  nonparametric  test  are  independence 
within  and  between  samples.  Field  test  trials  were  compared 
with  corresponding  Janus  scenarios.  Also,  the  test  FRE's  were 
pooled  across  trials  and  compared  with  the  pooled  Janus  FRE's. 
Table  VII  contains  the  results  of  the  tests.  The  test  fails 
to  reject  the  hypothesis  that  the  means  are  the  same  in  three 
cases  indicated  by  asterisks  in  Table  VII.  Since  the  p-value 
is  less  than  .05  in  the  other  13  cases,  the  hypotheses  that 
the  means  are  the  same  between  the  test  and  Janus  are 
rejected.  For  the  three  cases  where  the  Mann-Whitney  test 
fails  to  reject  the  null  hypothesis,  the  power  cannot  be 
calculated.  Closer  analysis  of  these  three  cases  was 
conducted  using  the  Two-Sample  t-test  as  an  approximate  data 
analysis  procedure.  This  test  requires  the  additional 
assumption  of  normality  of  the  samples,  but  is  generally 
considered  to  be  robust  with  respect  to  this  assumption.  The 
advantage  of  using  the  t-test  is  that  the  power  of  the  test 
can  be  calculated. 
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Table  VII   MEAN  COMPARISONS 


Mann-Whitney  Tests 
Test  Sample  Size/Mean;   (p-value) ;   Janus  Sample  Size/Mean 


FIRERvsTGT/ 
TRIAL 


Ml  vs  FST 


Ml  vs  BMP 


FST  vs  Ml 


FST  vs  CFV 


L099B 


LIOOB 


L112B 


L122B 


L123B 


22/1619m 
**(.5462) 
49/1639m 


8/1141m 
**(.1311) 
16/  559m 


26/  641m 
(.0000) 
203/1919m 


11/  567m 
(.0000) 
128/1937m 


>>V   ,  '/ 


25/1415m 

(.0000) 

72/2354m 


10/1045m 

(.0001) 

62/2277m 


tmWWWimiMIMMI^IHMMMIMMMIMMI 


15/  282m 

(.0000) 

56/1370m 


6/  467m 
(.0031) 
62/  975m 


**.*,»•* jX^^jT : 


18/  550m 
(.0000) 
117/2619m 


12/  852m 

(.0000) 

63/2788m 


>  s^.lt 


wtf^MMdA^dtfttUM^^i^tiirf^iUi^MUfa 


L125B 


13/  873m 

(.0354) 

13/1355m 


8/  465m 
(.0010) 
42/1400m 


10/1082m 
**(.0869) 
21/1583m 


7/1132m 
(.0004) 
181/2621m 


POOLED 


119/  951m 

(.0000) 
510/2039m 


55/  774m 
(.0000) 
373/1858m 


**  Fail  to  reject  hypothesis 
that  means  are  equal  between 
Field  Test  and  Janus. 


For  the  three  cases  under  question,  the  Two-Sample  t-test 
agreed  with  the  Mann-Whitney  test.  Figures  3  through  5 
contain  the  power  curves  associated  with  the  three  t-tests. 
We  wrote  an  A  Programming  Language  (APL)  program  to  calculate 
the  data  for  the  power  curves  using  [Ref.  11:  p.  128].  The 
power  function  in  Figure  3  indicates  that  if  the  alternate 
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hypothesis  is  the  difference  between  means  is  500  meters,  then 
the  probability  of  accepting  this  hypothesis  when  it  is  true 
is  .20.  If  the  difference  is  1000  meters,  the  probability  is 
.64.  The  power  becomes  high,  .94,  when  the  difference  is  1500 
meters.  A  difference  of  1500  meters  is  very  large  considering 
the  range  of  the  tank  is  approximately  3000  meters.  Thus, 
even  though  the  Two-Sample  t-test  fails  to  reject  the  null 
hypothesis  that  the  means  are  equal,  the  power  of  the  test  is 
low.  Analysis  of  Figure  4  indicates  even  lower  power  than 
Figure  3.  At  a  difference  of  500,  1000,  and  1500  meters,  the 
power  is  .10,  .31,  and  .65  respectively.  Analysis  of  Figure  5 
indicates  low  power  also  with  values  of  .15,  .41,  and  .86  for 
differences  between  means  of  500,  1000,  and  1500  meters, 
respectively.  Out  of  16  comparisons,  in  13  we  conclude  that 
the  means  are  different.  We  have  shown  the  low  degree  of 
power  associated  with  the  three  tests  that  conclude  that  the 
means  could  be  the  same.  Overall,  the  means  are  significantly 
higher  in  Janus.  Possible  reasons  for  these  differences  are 
discussed  in  Section  E. 
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Figure  5   Power  Curve  (FST  vs  Ml) 

D.   COMPARISON  OF  DISTRIBUTIONS 

1.   Kolmogorov-Smimov  (K-S)  Tests 

The  Kolmogorov-Smimov  (K-S)  test  was  used  to  test  the 
hypothesis  that  the  distributions  of  FRE's  are  the  same 
between  the  test  and  Janus.  The  assumptions  for  this 
nonparametric  test  are  independence  within  and  between 
samples.  Table  VIII  contains  the  results  of  the  K-S  tests. 
The  sample  sizes  are  the  same  as  reported  in  Table  VII.  Out 
of  16  comparisons,  the  K-S  test  rejects  the  hypothesis  that 
the  distributions  are  the  same  in  every  case  but  one.  This  is 
strong  evidence  that  the  distributions  are  different. 
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Table  VIII 

DISTRIBUTION  COMPARISONS 


Kolmogorov-Smirnov  Tests 
(p-value) 


FIRERvsTGT/ 
TRIAL 


Ml   vs    FST 


Ml   vs    BMP 


FST   VS   Ml 


■,V""Jg!,'t^'^gT^P^J""!i!'W'D'l." 


FST   vs    CFV 


L099B 


.0375) 


.0310) 


,'j  1 1  tuy^wj  ■■■■*!■»■ 
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'iiiiiiiin Ill  III  mil  1 1 II I  III  1 1 Ill 
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.0000) 


.0000) 


r"'"  >  ff.. 
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L112B 
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.0000) 
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.0000) 
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iiiiiiiiniiiiiiiiiiiiiiiiiiiiiiii  iiiiniiiiiiiiiiiiimiiimiiiiiiiiiiiiiiiiii 


iiMiViiSn   '     1 


L123B 


.0000) 


.0000) 


L125B 


.0146) 


.0002) 


**     (.2059)  (.0019) 

'  jw  ;  <     .  i^vi.4  Ai.      •:      ".v  S.  V»  *^  ^^   <*  w 


POOLED 


.0000) 


.0000) 


s-Wc>'-;>"< 


^•W^X^'^ 


.V^i  .?*4xs!::?^^' 


^  -^  ••  ■*  ■'.  <^  •••■■* 


**Fail  to  reject  hypothesis 
that  distributions  are  the  same 
between  Field  Test  and  Janus. 


2.   Empirical  Quantile-Quantile  Plots 

To  determine  how  the  distributions  differ,  empirical 
quantile-quantile  (Q-Q)  plots  were  drawn. 

a.  Method  of  Construction 

This  plot  is  constructed  by  plotting  the 
quantiles  of  one  empirical  distribution  against  another.  Each 
sample  of  FRE's  is  ordered  from  lowest  to  highest.  If  the 
sample  sizes  are  the  same,  the  ordered  samples  are  plotted 
against  each  other.  Since  the  sample  sizes  are  different  in 
the  data,  the  quantiles  from  the  larger  data  were  linearly 
interpolated  onto  the  smaller  data  set  [Ref,  12:  p.  55].  The 
ordered  smaller  sample  was  plotted  against  the  interpolated 
data . 
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b.  Interpretation 

The  line  Y=X  is  the  reference  for  the  empirical 
quant ile-quantile  plot.  If  the  plot  lies  along  the  line  Y=X, 
then  the  distributions  are  the  same.  If  the  plot  has  the  same 
slope  with  a  different  intercept,  then  the  distributions  are 
the  same  with  a  shift  in  location.  If  the  plot  is  a  straight 
line  with  a  different  slope,  then  the  variances  of  the 
distributions  differ.  Figures  6  and  7  contain  the  empirical 
quantile-quantile  plots  of  the  pooled  data  for  the  MlAl.  The 
points  do  not  lie  along  a  straight  line,  but  along  a  quadratic 
curve.  It  is  clear  that  Janus  produces  longer  engagement 
ranges  than  the  field  test  for  the  MlAl  up  to  3000  meters. 
Beyond  3000  meters,  the  ranges  are  longer  in  the  field  test; 
this  is  because  the  maximum  range  for  the  MlAl  in  the  Janus 
database  is  less  than  the  maximum  range  for  the  MlAl  in  the 
field  test.  In  the  field  test,  six  out  of  the  119  MlAl  FRE's 
were  beyond  3000  meters;  none  were  observed  beyond  3000  meters 
in  Janus.  Figure  8  contains  the  differences  between  field 
test  and  Janus  quantiles  plotted  against  the  Janus  quantiles. 
This  plot  shows  that  as  the  Janus  ranges  increase  to  2500 
meters,  the  difference  between  the  field  test  and  Janus  ranges 
increase;  the  test  ranges  vary  from  200  to  1500  meters 
shorter.   Beyond  2500  meters,  the  differences  decrease. 

Figure  9  contains  the  empirical  quantile-quantile  plot  for 
the  FST  versus  the  MlAl  in  Trial  L125B.  The  plot  appears  to 
be  a  straight  line  with  slope  equal  to  one  and  intercept  equal 
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to  -539.  Thus,  the  distributions  appear  to  be  the  same  with 
a  change  in  mean.  The  plot  shows  that  Janus  produces  ranges 
for  the  FST  that  are  approximately  539  meters  longer  than  the 
field  test.  It  would  be  inappropriate  to  conclude  that  Janus 
produces  FST  ranges  similar  to  the  field  test  in 
distributional  shape  based  on  analysis  of  only  one  trial 
consisting  of  ten  data  points. 

Overall,  the  K-S  tests  and  empirical  quantile-quantile 
plots  show  that  the  distributions  of  FRE's  within  and  across 
trials  are  different  between  Janus  and  the  field  test.  Janus 
does  not  generate  engagement  ranges  similar  in  mean  and 
distribution  to  the  field  test  data. 
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EMPIRICAL  QUANTILE-QUANTILE  PLOT 
MlvsFST  (6  TRIALS) 
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Figure  6  Q-Q  Plot  of  pooled  Ml  vs  FST  FRE's  (meters) 
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DIFFERENCES  BETWEEN  QUANTILES 
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Figure  9   Q-Q  Plot  of  FST  vs  Ml  FRE's  (meters) 
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E.   ANALYZING  THE  DIFFERENCES 

1.   Range  Versus  Time 

All  six  of  the  trials  have  Red  in  the  offense  while  Blue 
is  in  the  defense.  The  Red  forces  start  in  the  North  and  move 
toward  the  Blue  forces  in  the  South.  In  general,  as  time 
increases,  the  engagement  ranges  should  decrease  because  the 
opposing  forces  move  closer  to  each  other.  Tanks  remaining  on 
high  ground  may  get  long  range  shots  throughout  the  battle  or 
blue  forces  positioned  forward  may  engage  early,  but  most  of 
the  engagement  ranges  should  get  shorter  as  the  battle 
progresses.  A  correlation  analysis  of  range  and  time  for  the 
pooled  Janus  and  field  test  data  revealed  that  range  is 
negatively  correlated  with  time.  Since  both  Janus  and  field 
test  data  support  the  idea  that  range  decreases  as  time 
increases,  why  do  the  ranges  differ? 

Clues  about  why  Janus  produces  longer  engagement  ranges 
than  the  field  test  are  found  by  plotting  range  versus  time 
for  a  trial.  Figures  10  and  11  are  range  versus  time  plots 
for  trial  L112B.  The  field  test  version  of  this  trial  had 
four  blue  tanks  positioned  North  engage  Red  tanks  ten  minutes 
into  the  battle  at  a  range  of  400  to  800  meters  and  then 
withdraw  to  the  remaining  blue  forces  in  the  South.  The  main 
battle  occurred  at  50  minutes  at  a  range  of  1000  to  1800 
meters.  A  blue  tank  positioned  on  high  ground  engaged  at  4000 
meters  near  the  end  of  the  battle. 
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The  Janus  version  of  this  trial  had  the  four  blue  tanks 
engage  two  minutes  into  the  battle  at  a  range  1100  to  3  000 
meters.  The  main  battle  occurred  at  35  minutes  at  a  range  of 
2300  to  2800  meters.  Since  the  maximum  range  of  the  MlAl  in 
Janus  is  less  than  4000  meters,  the  4000  meter  engagements 
observed  in  the  field  test  were  not  possible  in  Janus.  Janus 
accurately  represented  the  flow  of  the  battle.  The  only 
difference  is  that  Janus  engaged  earlier  at  longer  ranges. 

An  interview  with  one  of  the  tank  company  commanders  from 

the  field  test  indicated  that  most  of  the  tank  engagements 

occurred  at  close  range  mainly  because  of  three  reasons. 

The  players  are  required  to  wear  laser  safe  goggles  that 
restrict  vision  to  varying  degrees. 

The  heavy  amount  of  dust  stirred-up  by  mechanized  vehicles 
obscures  targets. 

The  undulations  in  the  terrain  restrict  line-of-sight  beyond 
about  1000  meters. 

The  tank  company  commander  added  that  players  did  not  wait  for 

targets  to  enter  engagement  areas  before  firing.  Of  the  above 

reasons,  only  the  protective  goggles  would  not  be  a  factor  in 

actual  combat.  Even  if  the  players  were  not  required  to  wear 

goggles,  the  other  two  reasons  are  sufficient  to  support  short 

range  tank  engagements  at  Fort  Hunter  Liggett.   Janus  does 

represent  dust  and  terrain.   The  50  meter  terrain  resolution 

used  in  this  simulation  evidently  does  not  represent  the 

undulations  and  extent  of  vegetation  the  players  actually  face 

on  the  ground. 
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2.   Number  of  Rounds  Fired  in  a  Multiple  Engagement 
a.  Field  Test  Data 

Recall  that  we  defined  a  multiple  engagement  as  a  tank 
firing  at  the  same  target  within  the  inter-fire  time.  There 
is  evidence  in  the  field  test  data  that  the  number  of  rounds 
a  tank  fires  in  a  multiple  engagement,  N,  is  a  geometric 
random  variable  with  p  estimated  to  be  ,7391.  The  geometric 
random  variable,  N,  records  the  trial  number  of  the  first 
success.  Each  trial  has  the  same  probability  of  success,  p. 
For  our  model,  a  trial  is  the  tank  firing  a  round,  and  a 
success  occurs  when  the  tank  stops  firing  at  the  target  within 
the  inter- fire  time.   A  success  occurs  when, 

1.  the  target  is  hit, 

2.  the  firer  is  hit,  or 

3.  line  of  sight  no  longer  exists  between  the  firer  and  the 
target . 

Thus,  the  probabilities  that  a  tank  fires  N  rounds  during  a 

multiple  engagement  are  estimated  to  be, 

PiN=l)  =  .7391(1-. 7391)°  =  .7391  , 
P{N=2)  =  .7391(1-. 7391)1  =  .1928  , 
P(N=3)  =  .7391(1-. 7391)2  =  .0503  , 

and 

P(N^4)    =  1-  [P(N=1)    +  P(N=2)    +  PiN=3)]    =  .0178  . 


42 


Evidence  for  this  geometric  fit  was  obtained  using  the  Chi- 
Square  Goodness  of  Fit  test  shown  in  Table  IX  with  the  pooled 
MlAl  engagement  data.  The  estimate  for  the  parameter,  p,  was 
obtained  from 


y.  87  (1)  +24  (2)  ->-6  (3)  +2  (4)  _^    3^29 

119 

and 


fi-^- 


X     1.3529 


=.7391 


The  hypothesis  was  not  rejected  because 

P(X%  >  .06514)  =  .9680  >  .05 

Table  IX   CHI-SQUARE  GOODNESS  OF 
FIT  TEST 


Hq:  #  Rounds  per  Multiple  Engagement  is  Geometric 

#  Rounds  per 
multiple  engag 

Observed 

Expected 

[(0-E)2]  /  E 

1 

87 

87.95 

.01032 

2 

24 

22.95 

.04832 

3 

6 

5.99 

.00003 

>  4 

2 

2.11 

.00647 

Total 

119 

119 

.06514 

degrees  of  freedom  =  #cells  -  #parameters  estimated  -1=2 
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To  determine  a  hit  for  each  engagement  during  the 
field  test,  the  computer  compares  a  random  number  with  a 
probability  of  hit  contained  in  a  database.  Since  the 
probability  of  hit  increases  as  the  range  decreases,  the  high 
estimated  probability  of  success,  p  =.7391,  indicates  that 
field  test  engagement  ranges  are  short.  Because  loss  of  line 
of  sight  also  results  in  a  "success",  the  high  probability  of 
success  also  indicates  that  line  of  sight  between  the  firer 
and  target  is  frequently  broken. 

b.  Janus  Data 

Analysis  of  the  Janus  data  revealed  that  the  number  of 
rounds  fired  in  a  multiple  engagement  is  not  a  geometric 
random  variable.  The  estimated  probability  of  success  was 
lower  than  the  field  test,  p  =.4966,  and  the  null  hypothesis 
of  fit  to  a  geometric  distribution  was  rejected  with  a 
significance  of  .0442.  The  Janus  data  do  not  fit  a  geometric 
distribution  because  there  were  several  observations  in  the 
tail  of  the  geometric  distribution;  the  Janus  data  contained 
three  multiple  engagements  consisting  of  11,  14,  and  15 
rounds.  Seven  percent  of  the  multiple  engagements  contained 
five  or  more  rounds;  the  highest  number  of  such  rounds  fired 
in  the  field  test  data  was  four.  To  resolve  an  engagement, 
Janus  compares  a  random  number  with  a  probability  of  hit 
contained  in  a  database.  Since  the  probability  of  hit 
decreases  as  the  range  increases,  the  repetitive  firing  by 


44 


Janus  indicates  that  tanks  engage  at  longer  ranges  and  that 
line  of  sight  is  frequent  and  of  longer  duration. 
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V.   CONCLUSION  AND  RECOMMENDATIONS 

A.  CONCLUSION 

At  this  time,  Janus  should  not  be  accredited  for  post-test 
modeling  of  ground  vehicle  engagements  because, 

1.  statistically  significant  differences  in  tank  engagement 
ranges  exist  between  Janus  and  the  Line  Of  Sight-Forward- 
Heavy  Initial  Operational  Test;  and 

2.  the  test  data  were  insufficient  to  support  engagement 
range  analysis  of  the  Cavalry  Fighting  Vehicle  or  BMP. 

Six  of  the  50  field  test  trials  were  analyzed.   Janus 

consistently  generated  tank  engagement  ranges  200  to  1500 

meters  longer  than  those  observed  in  the  field  test.    In 

general,  Janus  accurately  represents  the  flow  of  the  battle, 

but  engages  targets  earlier  when  the  range  is  greater.   This 

suggests  that  line  of  sight  exists  in  the  model  when  it  does 

not  exist  in  the  field  test. 

B.  RECOMMENDATIONS 
1.   Field  Test 

The  field  test  data  contained  several  engagements  of 
unknown  range.  This  trend  was  apparent  in  all  the  trials  and 
especially  noticeable  with  the  BMP,  Cavalry  Fighting  Vehicle, 
and  Future  Soviet  Tank.  Improvements  in  instrumentation  are 
necessary  to  increase  the  fraction  of  engagements  that  have 
engagement  ranges  in  the  data. 
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2 .   Janus 

Currently,  the  Defense  Mapping  Agency  (DMA)  prepares 
terrain  for  combat  simulation  models  down  to  12.5  meter 
resolution.  Recommend  analysis  of  engagement  ranges  in  Janus 
using  terrain  resolution  lower  than  50  meters.  This  may 
require  segmenting  the  battle  because  terrain  resolution  lower 
than  50  meters  does  not  cover  the  entire  playing  area  at  Fort 
Hunter  Liggett.  If  Janus  continues  to  generate  longer 
engagement  ranges,  the  terrain  database  should  be  analyzed  for 
agreement  with  the  actual  terrain. 

The  parameters  in  the  Janus  combat  systems  database 
should  be  verified  by  a  team  of  testers  and  modelers. 

The  model  contains  a  target  selection  algorithm  that 
permits  the  firer  to  continue  firing  at  the  same  target  until 
the  target  is  destroyed  or  moves  out  of  line  of  sight.  On  the 
surface,  this  appears  to  be  a  reasonable  model  of  real  world 
engagements.  However,  in  some  cases  during  the  simulation, 
the  firer  expends  a  large  portion  of  the  basic  load  at  the 
same  target.  This  continual  firing  does  not  occur  in  the 
field  test.  Modelers  should  analyze  this  algorithm  to  see  if 
modifications  are  appropriate. 

Janus  displays  great  potential  as  an  effective  tool  in 
the  post-test  modeling  phase  of  the  Model -Test-Model  concept. 
Additional  comparisons  should  be  conducted  to  enhance  both  the 
model  and  field  test. 
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APPENDIX  A:   FORTRAN  CONVERSION  PROGRAM 

PROGRAM  PLSTRN 

*  THE  MAIN  PROGRAM  TAKES  THE  TEC  POSITION  LOCATION  DATA 

*  WHICH  IS  BY  SECOND  AND  EXTRACTS  EVERY  MINUTE  FOR  JANUS 

*  USE.  JANUS  CAN  ONLY  HANDLE  50  NODES.   EACH  TRIAL  IS 

*  ABOUT  50  MINUTES  LONG,  SO  POSITION  LOCATION  DATA  FOR 

*  EVERY  MINUTE  RESULTS  IN  APPROXIMATELY  50  NODES  PER 

*  VEHICLE. 

*  THERE  ARE  TWO  REQUIREMENTS  FOR  THIS  PROGRAM.  ONE,  THE 

*  POSITION  LOCATION  DATA  FILE  FROM  THE  FIELD  TEST.  TWO, 

*  THE  EXACT  NUMBER  OF  VEHICLES  RECORDED  IN  THE  DATA. 

INTEGER  SS,  X,  Y,  NUMVEH 
CHARACTER  PID*4 ,TIME*5,FILE*9 

COMMON  NUMVEH 

PRINT  *,  'INPUT  FILE  (EG  L112B.PLS)  ?  (USE  APOSTROPHES)' 
READ  *,FILE 

OPEN (UNIT=10 , FILE=FILE , STATUS= ' OLD ' , RECL=69 , 
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C  CARRIAGECONTROL='LIST') 


PRINT* ,' ENTER  TOTAL  #  OF  VEHICLES  (INCL  AIRCRAFT)  IN 
C  TRIAL' 


PRINT* , 
PRINT* , 
PRINT* , 
PRINT* , 
PRINT* , 


TRIAL  L099B=63' 
TRIAL  L100B=62' 
TRIAL  L112B=67' 
TRIAL  L122B=66' 
TRIAL  L123B=61' 


PRINT* ,' TRIAL  L125B=67 


READ* , NUMVEH 

PRINT  *, 'PROGRAM  CONTINUING. . . ' 

OPEN (UNIT=11 , FILE= ' CHRONORD . DAT ' , STATUS= ' NEW ' 
C , FORM= • FORMATTED • ) 
5   READ(10,10,END=30)  TIME,SS,PID,X, Y 
10   FORMAT(10X,A5,1X,I2,1X,A4,1X,I5,1X,I5) 

IF(SS.EQ.O)  WRITE(11,20)  TIME,PID,X,Y 

20   F0RMAT(1X,A5, ' : 00 ' , 2X, A4 , 2X, 15, 2X, 15) 

GO  TO  5 
30   CONTINUE 

CLOSE (UNIT=10) 

CLOSE (UNIT=11) 
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PRINT* , ' FINISHED  CONVERTING  PLS ' 
PRINT*, 'PROGRAM  CONTINUING. . . • 

CALL  FOURFILE 

END 


SUBROUTINE  FOURFILE 

C*****THIS  SUBROUTINE  CREATES  FOUR  FILES — 
C*  *  *  *  * ( NTCM0VE9  9  9 . DAT , NTCR0UT9  9  9 . DAT , NTCPLAY9  9  9 . DAT , 
C*****NTCKILS999.DAT)  WHICH  ARE  USED  BY  INITNTC  TO  RUN  JANUS. 
C*****ADDITIONALLY,  BADGRID999.DAT  CONTAINS  ALL  THE  GRIDS  FROM 
C*****THE  TRIAL  THAT  WILL  NOT  FIT  ON  A  50X50  JANUS  MAP. 

INTEGER  LPN,X,Y,NTCTYPE,I,NUMVEH,J 

CHARACTERDATE*9 , TIME*8 , TECTYPE*2 , SIDE*1 , PID*3 , BOGUS*64 

LOGICAL  WRITEPLAY 

COMMON  NUMVEH 

OPEN (UNIT=10 , FILE= ' NTCR0UT999 . DAT • , STATUS= • NEW • ) 
OPEN (UNIT=11 , FILE= ' NTCMOVE999 . DAT ' , STATUS= ' NEW ' ) 
OPEN (UNIT=36 , FILE= • NTCPLAY999 . DAT ' , STATUS= • NEW ' ) 
OPEN (UNIT=37 , FILE= •NTCKILS999 . DAT ' , STATUS= 'NEW' ) 
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C*****NTCKILS999.DAT  IS  A  FILE  WITH  HEADINGS  ONLY.   IT  IS  NOT 
C*****NECESSARy  TO  RUN  JANUS,  BUT  INITNTC  LOOKS  FOR  THE  FILE 
C*****AND  WILL  TERMINATE  WITHOUT  IT. 

OPEN (UNIT=14 , FILE= ' BADGRID999 . DAT ' , STATUS= • NEW ' ) 

WRITE (14,*)  'THESE  GRIDS  HAVE  BEEN  DELETED  SINCE  ' 
WRITE(14,*)  'THEY  DO  NOT  FIT  ON  A  50X50  JANUS  MAP...' 
WRITE (14,*)  '  ' 


C*****NTCM0VE999.DAT  USES  CHRONORD.DAT 

OPEN (UNIT=13 , FILE= ' CHRONORD. DAT ' , STATUS= ' OLD ' ) 

WRITE (10,*) 

WRITE(11,*) 

WRITE(36,*) 

WRITE(37,*) 

WRITE (10, *) •routs_all  table' 

WRITE(11,*) 'move_all  table' 

WRITE (36, *) 'pdscr  table' 

WRITE ( 37, *) 'kills_all  table' 

WRITE(10,*)  '  ' 

WRITE(11,*)  '  ' 

WRITE(36,*)  •  • 

WRITE(37,*)  '  • 
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WRITE(10,1) 
WRITE(11,1) 

1  FORMATC :time' ,16X, ' :lpn  '  , 2X, ' : side' , 2X, • :pid' , 3X, ' : • , 
C'ptype' ,1X, ' :x' ,4X, '    :y' ,5X, • : •) 

WRITE (10, 2) 
WRITE (11, 2) 

2  FORMAT  (  '  : : : : • 

C» • .^...^ • «_.«.« • ...... • t \ 
/    *  *  *  *    / 

WRITE(36,33) 
WRITE(37,34) 

33  FORMAT( ' :lpn' ,3X, ' :pid' ,3X, ' :side' ,2X, • :org' ,17X, • :ptype 
C  •) 

34  FORMAT (' :tlpn' ,1X, • :tpid' ,1X, • :side' ,1X, ' : result' , 
C  ' :time' ,16X, ' :tx' ,3X, • :ty' ,3X, • :flpn' ,2X, ' :fpid' 
C,2X, • :fwpn' ,2X, • : fx' , 3X, ' : fy ' ,3X, ' :fy',3X, ' : f rat   : ') 

WRITE(36,*) •  • 
WRITE (37,*)'  ' 

C*****DUMMY  DATE 

DATE='12  APR  90' 


C*****CREATING  NTCMOVE999.DAT 

LPN=0 
7   LPN=LPN+1 
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IF(LPN.EQ.NUMVEH+1)  LPN=1 

READ (13,20, END=4  0 )  TIME , TECTYPE , PID , X , Y 

10   CONTINUE 

IF ( TECTYPE. EQ. 'AH')  THEN 
NTCTYPE=22 
SIDE='B' 
ELSEIF (TECTYPE . EQ . ' BR ' )  THEN 
NTCTYPE=29 
SIDE='B' 
ELSEIF (TECTYPE . EQ . ' FT ' )  THEN 
NTCTYPE=1 
SIDE='B' 
ELSEIF (TECTYPE . EQ . ' LO ' )  THEN 
NTCTYPE=10 
SIDE='B' 
ELSEIF (TECTYPE . EQ . ' OH ' )  THEN 
NTCTYPE=26 
SIDE='B' 
ELSEIF (TECTYPE . EQ . » CC • )  THEN 
NTCTYPE=14 
SIDE='B' 
ELSEIF ( TECTYPE. EQ. 'AG')  THEN 
NTCTYPE=25 
SIDE='B' 
ELSEIF (TECTYPE . EQ . ' BM ' )  THEN 
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NTCTYPE=3 

SIDE='0' 
ELSEIF (TECTYPE . EQ . ' HP ' )  THEN 

NTCTYPE=27 

SIDE='0' 
ELSEIF (TECTYPE. EQ. 'TT')  THEN 

NTCTYPE=1 

SIDE='0' 
ELSEI F( TECTYPE. EQ. 'TV')  THEN 

NTCTYPE=20 

SIDE='0' 
ELSEIF (TECTYPE. EQ. 'FF')  THEN 

NTCTYPE=25 

SIDE='0' 
ELSEIF (TECTYPE. EQ. 'TH'  )  THEN 

NTCTYPE=23 

SIDE='0' 
ELSE 

PRINT* , ' DO  NOT  HAVE  A  NTCTYPE  MATCH  FOR  TECTYPE 
C    • , TECTYPE 

PRINT*,  'HAVE  ASSIGNED  IT  A  NTCTYPE  OF  0  (ZERO)  AND  PUT 
PRINT*, 'ON  THE  BLUE  SIDE' 

PRINT* ,  '  PROGRAM  CONTINUING • 

NTCTYPE=0 
SIDE='B' 
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ENDIF 

IF( (X. GT. 50000. AND. X.LT. 65000) .AND. (Y.GT. 7  3  000 
C   .AND.Y.LT. 88000) )   THEN 

WRITE(11, 30)  DATE, TIME, LPN, SIDE, PID,NTCTYPE,X,Y 
ELSE 

WRITE  (14,2  0)  TIME, TECTYPE, FID, X,Y 
ENDIF 

GO   TO    7 


2  0   F0RMAT(1X,A8,2X,A2,A3,1X,I5,2X,I5) 
30      FORMATC:     '  ,  A9  ,  IX,  A8  ,  '  :  •  ,  13  , 

w  f   O  A  /   *   f  /\^  f   O  A  ^   *   /  fL^  f  tj  A  ^   •   •  ^  ^ 

40      CONTINUE 


CLOSE (UNIT= 11) 
CL0SE(UNIT=13) 
CLOSE (UNIT=14) 

C*  *  *  *  *  * CREATING   NTCR0UT9  9  9 . DAT 

OPEN (UNIT=81 , FILE= ' NTCMOVE999 . DAT • , STATUS= • OLD ' ) 
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1=1 

95   DO  76  J=l,5 

READ (8 1,82)  BOGUS 

82  FORMAT ( IX, A6 4) 
76   CONTINUE 

WRITEPLAY= . TRUE . 

83  READ(81,84,END=66)  DATE, TIME, LPN, SIDE, PID,NTCTYPE,X, Y 

84  FORMAT(2X,A9,lX,A8,2X,I3,4X,Al,6X,A3,4X,I2,5X,I5,2X,I5) 

IF(LPN.EQ.I)  THEN 

WRITE (10,30)  DATE , TIME , LPN , S IDE , PID , NTCTYPE , X , Y 
IF(WRITEPLAY)  THEN 
C******CREATING  NTCPLAY999.DAT 

WRITE (36, 85)  LPN, PID, SIDE, NTCTYPE 
WRITEPLAY= . FALSE . 
ENDIF 
ENDIF 
GO  TO  83 
66   CLOSE (UNIT=81) 

OPEN (UNIT=81 , FILE= ' NTCM0VE999 . DAT ' , STATUS= ' OLD • ) 

85  FORMAT (• : • , 13 , 3X, ' : ' , A3 , 3X, ' : ' ,A1,5X, • : • ,20X, • : • ,I2,4X 
C  ,':•) 
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1=1+1 

IF(I.EQ.NUMVEH+1)  GO  TO  99 
GO  TO  95 

99   CONTINUE 

WRITE (10, 2) 
WRITE (37,*) '  ' 

RETURN 
END 
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APPENDIX  B:   LIST  OF  ACRONYMS 

BMP  Soviet  Mechanized  Infantry  Vehicle 

BTL  Battle  File  from  field  test  data 

CFV  Cavalry  Fighting  Vehicle,  US  Army 

FDTE  Force  Development  Test  and  Evaluation 

FORTRAN  Formula  Translation  Computer  Language 

FRE  First  Range  of  Engagement 

FST  Future  Soviet  Tank 

IFT  Inter-Fire  Time 

INITNTC  Program  to  convert  NTC  battles  to  Janus  format 

lOT  Initial  Operational  Test 

JAWS  Janus  Analyst  Workstation 

LOS  FH  Line  Of  Sight  Forward  Heavy  Air  Defense  System 

MlAl  US  Army  Abrams  tank 

MTM  Model  Test  Model 

NTC  National  Training  Center 

NVEOL  Night  Vision  Electro  Optical  Laboratory 

PLS  Position  Location  System 

RMS  Range  Measuring  System 

TEC  TEXCOM  Experimentation  Center 

TEXCOM  Testing  and  Experimentation  Command 

TOF  Time  Of  Flight 

TRAC  Training  and  Doctrine  Analysis  Command 
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